Social robot with environmental control feature

ABSTRACT

A method and apparatus for controlling a social robot includes operating an electronic output device based on social interactions between the social robot and a user. The social robot utilizes an algorithm or other logical solution process to infer a user mental state, for example a mood or desire, based on observation of the social interaction. Based on the inferred mental state, the social robot causes an action of the electronic output device to be selected. Actions may include, for example, playing a selected video clip, brewing a cup of coffee, or adjusting window blinds.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/US17/60951, filed Nov. 9, 2017, which claims priority pursuantto 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No.62/420,492 filed Nov. 10, 2016, each of which are hereby incorporated byreference in their entireties.

FIELD

The present disclosure pertains to social robots, and to methods forcontrolling social robots.

BACKGROUND

Various humanoid, non-humanoid, and combined humanoid/non-humanoidrobots are known in the robotic arts. A “robot” is a programmable,autonomous or semi-autonomous machine programmed to react to sensedenvironmental inputs and thereby interact with some independent aspectof its environment. Non-humanoid robots are often designed to performspecific tasks that do not involve interacting with humans, for example,assembly-line manufacturing, or automated handling of cargo and thelike, that involve interacting with inanimate objects only. A humanoidrobot is constructed to resemble a human, including featuresrecognizable by people as a torso, arms, legs, a head, a face, and soforth. Realistic humanoid robots are complex and expensive to build, andaccordingly are not common. A combined humanoid/non-humanoid robotincludes a least one human-recognizable feature, combined with featuresthat are not humanoid. For example, a smartphone that includes a speechresponse feature in response to human speech is a sort of combinedhumanoid/non-humanoid robot. Another example is a virtual robot used tointeract with human users in virtual environments such as online socialnetworking sites or video games, appearing to be another user.

Both humanoid and combined humanoid/non-humanoid robots are often usedas social robots. As used herein, a “social robot” is an interactivemachine designed to evoke an emotional or social behavioral responsefrom a human or other social animal that interacts with it, byexhibiting behaviors that are recognizably social in nature for thehumans or other social animals with which the social robot isinteracting. Behaviors that are recognizably social in nature mayinclude, for example, vocalization, speech, body language (e.g.,posturing), facial expression, and other behaviors as described hereinbelow.

Like any other computing machine that interacts with a human user, arobot may be capable of leveraging the processing power and vast amountof data in a computer of a computer network to provide content that auser is interested in. With the advent of the “Internet of things,” moreappliances and devices are connected to wide-area networks, and can becontrolled using a network terminal. Network terminals are designed torespond to user commands and can be programmed to detect and record userpreferences for identifying advertising or other content likely to be ofinterest to the user. However, network terminals are not designed tomimic human social behavior; they are instead optimized for direct userinteraction with special-purpose applications generally designed foraccessing various types of content: e.g., web pages, video, databases,and so forth. In contrast, social robots by definition include featuresdesigned for social behavior, for example, displaying mood or emotion.This social function of robots is not viewed as useful for the sorts offunctions that computer terminals perform; indeed, it might seem sillyto use a social robot both for social and terminal functions. The‘terminal potential’ of social robots, however that might be imagined,is for the most part wasted.

It would be desirable, therefore, to provide methods for utilizingsocial robots in less limited ways, and to provide social robots thatleverage both their social and terminal potentials to provide new,synergistic and unexpected benefits to users.

SUMMARY

This summary and the following detailed description should beinterpreted as complementary parts of an integrated disclosure, whichparts may include redundant subject matter and/or supplemental subjectmatter. An omission in either section does not indicate priority orrelative importance of any element described in the integratedapplication. Differences between the sections may include supplementaldisclosures of alternative embodiments, additional details, oralternative descriptions of identical embodiments using differentterminology, as should be apparent from the respective disclosures.

As used herein, a ‘social action’ is an action performed for the purposeof eliciting a social response and is therefore performed with theawareness of the person from which a response is desired. Some actions(e.g., facial expressions) are purely social at least when performed bya robot, because there is no reason for a robot to perform the actionsexcept to elicit a social response. Other actions (e.g., filling a glassof water for a thirsty user, or turning on an appliance) may have both asocial purpose and other objectives (e.g., placing clean water in aglass without spilling. In the present application, a social action by arobot is any action for which the robot tracks a user's social response,and for which an extent of correlation between the robot's social actionand the user's social response determines further robot action. Socialactions by a robot may include, for example, facial expression, bodylanguage, including posturing of humanoid elements (e.g., head, arms,etc.), speech, vocalization, movement, gesture, touch, providing anobject or substance to the user, removing an object or substance fromthe user, turning another machine on or off, controlling operation of amachine or appliance to elicit a social response, playing recorded audioor audio-video content, or any other action intended to communicate asocial message or signal to the robot's user.

A social robot may be implemented as a real machine with the power tomove at least part of its robotic body in a real environment. Inaddition, a social robot may be implemented as a virtual machine thatcan be expressed only in a virtual environment, such as, for example, asa robotic character in a video game environment, in an immersive virtualreality environment, or in an augmented reality (AR) environment. Asused herein, the term “social robot” generally encompasses both real andvirtual social robots. When specifically pointing out one of theseclasses of social robots, the terms “real social robot” or “virtualsocial robot” will be used herein.

The social robot may include one or more processors coupled to at leastone circuit for driving components of the social robot, and a socialexpression component connected to the at least one circuit. The socialexpression component may include a human-recognizable expressive featurecontrolled by the one or more processors according to at least onesocial behavior algorithm. In an embodiment, the social expressioncomponent may include a video screen that displays an animated face, orexpressive parts of a face such as eyes, eyebrows, mouth or lips) of thesocial robot.

The social robot may further include a video camera connected to the atleast one circuit and positioned on the social robot for observing auser and/or the user's environment. The robot may also include amicrophone connected to the at least one circuit, positioned forreceiving sound from the user and/or the user's environment. The robotmay include, or may be coupled to, one of more sensors for measuring oneor more bodily states of the user, for example, pulse sensor, skintemperature sensor, gaze direction sensor, and so forth.

The social robot further includes a program memory, coupled to the atleast one circuit and to the one or more processors. The program memoryholds program instructions configured for execution by the one or moreprocessors, causing the social robot to perform operations including:interacting socially with the user by operating the social expressioncomponent, determining user behavior based at least in part on data fromthe video camera and the microphone, correlating the user behavior tosocial behavior of the social robot, generating a data signal based onthe correlating, and transmitting the data signal configured for causingan electronic output device or other electro-mechanical device (e.g. anelectronically activated valve used to fill a coffee pot) to respond toinformation encoded by the data signal.

In more detailed aspects, the instructions are further configured forcausing the social robot to perform deriving a current mental stateindicator based on the correlating, and causing selection of anidentifier of a prerecorded or online audio-video content clip based oncomparing semantic metadata associated with the identifier to thecurrent mental state indicator. For example, the social robot may derivean indicator for a “sleepy” mental state, and based on the time of day,the sleepy indicator, and the user's behavior patterns or past contentselections, choose an identifier for a lively, rousing audio and/orvideo clip. In an aspect, for example, the memory of the social robotholds identifiers for prerecorded or online audio-video content clipseach associated with semantic metadata pertaining to corresponding onesof the content clips, and the instructions are further configured forcausing the social robot to perform executing a content-selectionalgorithm, thereby the causing the selection of the identifier. In analternative, the instructions are further configured for causing thesocial robot to perform transmitting an indicator of the internal userstate to a remote server and receiving the identifier from the remoteserver. In the alternative embodiment, selection of the content clip isperformed by the remote server. In an alternative embodiment, a remoteserver may also supply the selected content.

In another aspect, determining the current mental state indicatorincludes determining an indicator of a user mood (happy, sleepy, bored,etc.) or an indicator of a user desire (wants quiet, wants to beentertained, etc.), using a behavior analysis algorithm. The algorithmmay compare a current social interaction with past social interactionsto determine a mental state. For example, if the social robot emitsspeech or a polite sound for obtaining attention, and in response isignored by the user while detecting that the user is active, the robotmay conclude that the user is “distracted” or “busy.”

In some embodiments, the electronic output device is, or includes, anaudio-video output device, and the instructions are further configuredfor causing the social robot to perform configuring the data signal sothat the prerecorded or online audio-video content clip is output by theaudio-video output device. For example, the social robot may cause avideo screen located on the robot itself to display the video contentclip. The video screen may also be used to display an animated face ofthe social robot. In an alternative, the video screen may be a separatecomponent, located on the robot or as an independent appliance (e.g., atelevision or mobile phone of the user, or part of an augmented realityheadset or glasses worn by the user).

In an alternative, or in addition, the electronic output device orelectro-mechanical device is a home appliance, for example, a coffeemaker, a faucet, a motorized window blind, an electronic lock, a homeaudio system, or a home theater system. The instructions may furtherinclude configuring the data signal so that it causes the home applianceto respond. For example, the information encoded by the data signal mayinclude operating instructions for the home appliance. Thus, the socialrobot may analyze the social behavior of the user and anticipate orrespond to such needs as the social robot is able to fulfill, by itselfor in cooperation with ancillary appliances.

In another aspect, the electronic output device or electro-mechanicaldevice may be a motor vehicle or vessel, for example, a self-drivingcar. In such cases, the instructions may include configuring the datasignal so that it causes the motor vehicle or vessel to respond, such asby encoding vehicle operating instructions. For example, the socialrobot might control the audio system of the vehicle in response todetecting a social signal from the user, start or power on the vehicleif anticipating that the user will soon want to go for a ride, or directa self-driving vehicle to transport the user to a specified destination,wherein the robot selects the destination at least in part on a currentmental state determined based on social interaction between the user andthe social robot.

The social robot may include a set of modules under common control witha defined relationship to one another, wherein the set as a whole isdesigned to interact socially with a human (or other, e.g. pet dog)companion. For example, a Jibo™ is a passive torso with a swivelinghead, video-screen face, ears (audio input) and static (non-animated)speech output. A smartphone provides an example of another type of robotconsisting of a passive frame, video-screen face, ears (audio input) andstatic (non-animated) speech output. Robots such as Asimo™ are morecomplex, have non-video facial features, and means of locomotion and forarticulating extremities. A decision engine controls output of thesocial robot in response to user actions such as speech, gaze direction,facial expression, body language, and other social cues. The decisionengine may be configured to exhibit a particular robot personality ormood, which may vary based on environmental factors or user commands.Further details regarding controlling a robot to express a certainpersonality may be as described in U.S. Provisional Patent ApplicationNo. 62/367,335, filed Jul. 7, 2016 (the “'335 Application”), which isincorporated herein in its entirety.

The '335 Application disclosed methods for selecting a social robot'sresponse based on a personality profile in conjunction withstimulus-response pair detection. A robot is programmed to match astimulus (including a context of the stimulus) with a response through apersonality filter or profile. In an aspect, a robot response mayinclude generating a data signal for causing an electronic output deviceto respond to information encoded by the data signal, for exampleselecting a video clip to play for the user. In other words, thepersonality profile of a social robot may influence the selection ofcontent played or other actions based on the robot's personalityprofile, using methods as described in the '335 Application. Forexample, a robot with a ‘child’ personality may select differentaudio-video content for playing from a robot with an ‘adult’personality, given the same social interaction and context.

In some embodiments, a robot may use a ‘test’ program to identifyinteracting human or environments in which interaction is taking place.For example, a social robot might interact with humans using a testprogram. The test program need not take the apparent form of a test.Instead, the test program may be designed to be perceived as a one-timeset-up or configuration process, a game, or some other entertainingprocess that samples an interacting person's personality, while alsoentertaining them. For example, the test may sample the interactinghuman's voice, traits, mood, expressions or mannerisms, and storeidentifying characteristics in the person's profile for futurereference. An initial interaction setup or configuration process mightbe executed by the robot only initially when interacting with a human,or perhaps every time a session with the robot commences, so that therobot could determine the interacting human's age, sex, height andbuild, mobility (i.e. whether the human is able to move freely, orbedridden, or feeble, or . . . ), spoken language (since the robot couldbe conversant in several), or other factors. The robot may store theperson's profile, and thereby later recognize that person, e.g. byemploying facial recognition software, when that person is againinteracting with the robot. Using voice recognition software, once aperson has been identified, then even if her/his appearance changes soas to render the person unrecognizable visually, the robot may stillmake the identification based on voice.

Similarly, via object recognition or image analysis software, the robotmay determine characteristics of the space which the robot and humanoccupy (e.g. indoors, outdoors, in a small area or a large one, whattime it is, whether it's daylight or nighttime, etc.) so thatappropriate conversation and related behaviors could be best managed.For example, it might seem inappropriate to speak of how lovely thestars appear to be if it's broad daylight, and there's no need to say“Let's go for a walk” if the human can't move or is preparing forbedtime. In this manner, the robot may either mimic or counterpoint thevoice, traits, mood(s), expressions or mannerisms depending upon what iscircumstantially appropriate. If, for example, the robot later detectsat a subsequent meeting (for example by identifying through machinelearning or AI or consultation with templates or consultation withdatabases of examples) that the human is depressed, then it could setabout trying to lift the human's spirits by telling a joke, offering toplay a game, playing a favorite song or video clip, or othermood-lifting interaction. All of the above interactions may take placevirtually, i.e. with the robot and the human(s) in different places,interacting by way of some audiovisual system—e.g. via Skype.

In another aspect, operation of a social robot is integrated with alocal or remote database of content that is selectively shared with theuser of the social robot, in response to a current stimulus oranticipated future condition. For example, the database of content mayinclude video clips, audio-video clips, or audio clips that are indexedbased on their semantic content. The social robot may be configured tooutput a selected one of the clips for any social purpose. For example,if the social robot senses that its user is unhappy, it may select ahumorous clip from clips of the user's known favorite content orcharacters. For further example, if the social robot is commenting on asensed social situation, for example, to encourage the user to get someexercise, it might play a clip that is marked as appropriate for userswith personal or demographic characteristics that matches the currentuser and that in some way reminds the user to get up and do someexercise. Accordingly, users may be entertained by the context andselection of clips selected by the social robot's content selectionalgorithm. In more detailed aspects, and for further example, the robotmay:

learn what type of relevant content clips the current user or users arelikely to pay attention to, for example by identifying areas, genres,works (prose, poetry, pictures, videos, art, etc.) that interest thetarget user(s) by questioning or otherwise ‘testing’ the target users(s)for interest levels, and then audio-visually output the selected contentclips;

wirelessly send selected content clips (or have them sent from a remoteserver) to audiovisual display gear (e.g., speakers, a phone, a tablet,a PC, a smart TV, an AR headset or glasses, etc.) independent of therobot so that the target user(s) can see and/or hear the content clipsynchronized to a social interaction involving the social robot, or inresponse to an authorized user request;

command a wireless device such as, for example, the Amazon™ Echo™ andFire Stick™, to tune in or otherwise obtain relevant content for thetarget user(s), wherein the content is selected by the social robotbased on user/situational parameters or selected by the user.

The method and apparatus of the present disclosure may be embodied insocial robots of various types, and in computer networks used to supportoperation of social robots. For example, selection of relevant contentclips may be performed partly or entirely using cloud computingresources, with results shared with a local robot for use locally. Thetechnical description that follows enables all of the functionalabilities of a social robot that are summarized above.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify like elements correspondingly throughout thespecification and drawings.

FIG. 1 is a block diagram illustrating aspects of a system and methodfor a social robot that selects entertainment content or other responsebased on social interaction with a user.

FIG. 2 is a flow chart showing elements of a method by a social robotfor selecting recorded digital content or other response based on socialinteraction with a user.

FIG. 3 is a flowchart showing aspects of a method by a social robot fordetermining a symbolic output (e.g., a mental state indicator) based onsocial actions of a user.

FIG. 4 is a table illustrating aspects of social action data for use inconnection with the method of claim 3.

FIG. 5 is a flowchart showing aspects of a method by a social robot forselecting and generating an environmental control output based onsymbolic output indicating a mental state from a process such asdiagrammed in FIG. 3.

FIG. 6 is a table illustrating use of semantic metadata for selectingdigital content based on a mental state indicator.

FIGS. 7A and 7B are concept diagrams illustrating aspects ofquantifiable stimulus and response parameters, respectively, whichpaired together can be used to define a social behavior numerically.

FIG. 8 is a table illustrating aspects of a data structure forrepresenting a stimulus or response numerically.

FIG. 9 is a concept diagram providing a graphical illustration of a 4×6numeric stimulus value paired to a 4×4 numeric response value, whereinthe illustrated stimulus-response pair (a behavior) correlates to asingle 1×8 numeric mental state indicator for the behavior.

FIG. 10 is a flowchart showing aspects of a method and system forcharacterizing a stimulus, in real time for operation of a social robot,wherein the stimulus correlates to one or more responses based on one ormore personality profiles in use for controlling the social robot.

FIG. 11 is a block diagram showing aspects and components of a realsocial robot.

FIG. 12 is an isometric view showing an example of a real social robot.

FIG. 13 is a block diagram showing aspects and components of a virtualsocial robot.

FIG. 14 is combined block diagram and rendering showing an example of avirtual social robot.

FIG. 15 is a concept diagram illustrating allocation of a response tocoordinated real and virtual social robots.

FIG. 16 is a flow chart illustrating an example of a method foroperating a virtual social robot for producing an environmental controloutput based on social interaction with a user (or users).

FIGS. 17-18 are flowcharts illustrating further aspects of the method ofFIG. 16.

FIG. 19 is a block diagram, illustrating an example of a social robotconfigured for performing the method of FIG. 16.

DETAILED DESCRIPTION

Referring to FIG. 1, a social robot system 100 including a social robot102 for selecting digital content (e.g., recorded or online audio-videocontent) or other environmental control action is illustrated. It shouldbe appreciated that the social robot 102 may be adapted for selectingother environmental control functions in response to a user's socialbehavior. The system includes various modules and data structures thatmay be incorporated in hardware, software, firmware, or in somecombination of the foregoing. The hardware, software, firmware, orcombination may be implemented locally in the social robot itself, or inpart using computer processing power that is in communication with thesocial robot using a computer or data communications network.

The social robot 102 may comprise an “Internet of Things” (IoT) node ina system 100 that includes other IoT nodes, Local Area Network (LAN)nodes, or other computer networks. The social robot can take anenvironmental control action by sending a control signal to another nodein the system 100. As used herein, an “environmental control action”includes signaling to a machine or device that responds to the signalingby altering an environment of the user 150 who is interacting sociallywith the social robot 102. For example, a smart display device 130 mayreceive a signal from the robot 102, and in response play a selectedvideo clip that relates semantically to social signaling of the user 150that is detected by the social robot. A smart appliance 128 (e.g., acoffee maker or other beverage or food preparation device) may receive asignal from the robot 102, and in response brew or dispense a beverageor food for the user 150. A smart home controller 122 may receive asignal from the robot 102, and in response adjust the lighting in theuser's 150 environment by signaling to a smart lighting system 124, orcontrol playing of music or ambient sounds by signaling to a smart audiosystem 126.

The smart display device 130 may be, or may include, a smartphone,notepad computer, personal computer, smart TV, virtual reality headset,augmented reality headset, or other display device. The robot 100 maysend a signal from transmitter to the smart display device 130, directlyin a peer-to-peer transmission, or indirectly through a computer and/orcommunications network. The signal from the robot 102 to the displaynode 130 may identify a content video clip (for example, “clip #12345”)or digital graphic image that the robot 102 is requesting that thatsmart display device 130 play or display. The smart display device 130may be located within sight of the user 150 who is interacting sociallywith the social robot 102, for example, in the same room. The requestedvideo clip or image may reside in a memory of the display node 130 ormay be requested by the device 130 or robot 102 from a data store, forexample, from the server 134 via a router 132 and wide area network(WAN) 136, or from an online source. In addition, or in the alternative,the social robot may play or display the requested video clip or digitalimage using an internal display device 110.

Accordingly, a purpose of the social interaction between the user 150and the robot 102, and processing performed by the robot 102 asdescribed in more detail herein, includes pleasing the user 150 bydisplaying content (e.g., videos or still images) playing audio content,or otherwise adjusting the user's environment in a way that complementsthe user's moods or desires, in part by selecting actions that surprise,delight, or otherwise engage the user. In other words, the user does notneed to request that specific content be played, or specific actions betaken to adjust the environment, for example. Instead, the robot 102 orother element of the system 100 (e.g., the server 134) selectsappropriate content based on social interaction of the user 150 with therobot 102. The social action may be completely non-verbal, and need notbe a response to any explicit, verbal request or command by the userthat requested action be taken. Instead, operation of the robot 102 andsystem 100 anticipates actions that will be pleasing to the user orfulfill some function in the social interaction between the robot 102and user 150.

A real (as opposed to virtual) social robot 102 may includeelectro-mechanical components including an electrical circuit configuredas a computer housed in an exoskeleton or housing. The circuit mayinclude a central processing unit or units 104, operatively coupled tomotor driver controller driving a motor 108, which moves a loadcomprising a mechanism connected to a head or body portion of the socialrobot 102, causing the head or body portion to move in a sociallysignificant way. For example, the motor 108 or set of motors may controlmovement of a head relative to a torso, such as by turning the headaround one or more independent axes or moving a robot torso relative tothe ground using wheels, treads, legs, or other mechanisms. Theprocessor 104 determines responses and provides motion control signalsto the motor driver 108. The motion control signals cause the motor 108to move the connected body part in conformance with the responseselected by the processor 104.

The robot 102 may contain other components for computing in addition tothe processor 104. For example, the processor circuit may be coupled toa user interface component (not shown), such as for example a keypad ortouchscreen, for entry of user controls. The processor circuit may becoupled to a display component 110, for example a touchscreen or otherdisplay. The display 110 may be used for a user interface, to displayexpressive features of the social robot (e.g., a face), or for both. Theprocessor circuit may be coupled to one or more sensors 114, 116, 120configured for sensing one or more environmental conditions of theenvironment around the robots, or of objects that the robot interactswith. The one or more sensors 120 may include, for example, a camera114, light sensor, microphone 116, temperature sensor, pressure sensor,accelerometer, touch sensor, proximity sensor, or other sensing orInternet of Things (IoT) microelectromechanical sensor (MEMS) device.The processor circuit may be coupled to a communications port 105 forcoupling to a computer network, or to a data communication network forreceiving data and program instructions. The processor circuit mayfurther be coupled to an audio output transducer (not shown) foroutputting robot vocalizations. The processor circuit may further becoupled to a volatile random access memory device 106 for storage ofprogram instructions and data.

The '335 Application referenced above describes a robotic behaviormapping subsystem used in advance of robot operation, to map personalbehaviors to social robot behaviors where possible, and to correlate allsocial behaviors of the robot to a personality scale used for definingthe robot's personality profile. The real-time social robot behaviormodule operates in real time when the social robot is interacting withits live user or users, causing the social robot to express one or moreof the personalities. A personality profiling subsystem receivesbehavioral data for a character or person, and analyzes the data basedon a predefined or adaptable personality scale. As used herein,“behavioral data” is limited to data that describes or defines sets ofstimulus-response pairs, including but not limited to statistical dataregarding relative frequency of occurrence of any particular pair.Stimuli and responses may be described at any desired level of detail,including different levels of detail. For example, a stimulus andresponse may be generic, such as a stimulus of ‘being ignored’ and aresponse of ‘seeking to be noticed.’ For further example, either or bothof the stimulus may be much more specific, such as a stimulus of hearing“How are you?” paired with a response of “What's it to you!?” Whilebehavior may be characterized in any useful manner, it may be desirableto choose stimuli and responses for profiling a character's personality,that can readily be mapped to stimuli that the social robot is capableof detecting, and responses that the social robot is capable ofperforming. In addition, the behavior should be, or be capable of beingcorrelated to, a social function of the robot. Behavior detection andanalysis as described in the '335 Application may be used for assessinguser social interaction for purposes of selecting content.

For example, character analysis may be used for mood detection. Acharacter analysis module may produce two outcomes from an analysisprocess: a list of characteristic social behaviors, and a set of one ormore personality profiles, each correlated to a distinct mood of theuser. A ‘mood’ may be thought of as a ‘personality mode’; in technicalterms each mood is a distinct set of personality trait values that canbe observed for continuous periods and to emerge and/or dissipate withat least some predictability in response to one or more mood triggers. Acharacter analyses process may discover a user's moods by groupingstimulus-response data over different continuous time periods, matchingsimilar behavior over different periods, and identifying triggers fortime periods in which a distinct mood is observed. The robot 102 maybase content selection or determination of other actions in part on auser mood that the robot observes using a process as outlined above.

FIG. 2 summarizes aspects of a social interaction analysis and responsemethod 200 that may be performed by the social robot 102. At 102, whichmay occur near the outset of a new session, the social robot identifiesa user that is interacting with it. For example, the robot may usefacial recognition via a camera and image analysis, voice recognitionvia a microphone and audio analysis, or other biometric identity factorsto identify a user that is interacting with the social robot. If theuser is not recognized as a particular individual, the social robot mayclassify the user based on observed demographic clues (e.g., size, age,gender) and retrieve at 208 a default preference profile for theobserved demographic type. If the robot recognizes the user, the robotmay retrieve a user preference profile at 210. A preference profile maybe, or may include, a digital data structure holding a set of symbols(e.g., binary numbers). These symbols indicate in any suitable way userpreferences for certain types of recorded content, favorite charactersor genres, and other preferences that relate to content or actionselection by the robot.

At 212, the social robot performs an action (i.e., a ‘stimulus’ from theuser's point of view). In addition, during and immediately after theaction, the robot collects sensor data that indicates a user response tothe robot's action. For example, the robot may move relative to the user(e.g., approach, move away, or turn to face) and emit a vocalization ingreeting. The robot may also generate a facial expression using a facecomponent. At 214, the robot receives and analyzes sensor data todetermine a user response. For example, the robot may receive andanalyze audio or video input to determine how, if at all, the user hasverbally responded, and any available emotional subtext provided byfacial expression or body language that is imaged by the robot's cameraor cameras.

The robot stimulus may be matched to a user response, and the resultingstimulus-response pair (representing a social action) is checked againsta data structure of recognized matched pairs. If a stimulus-responsepair is recognized by matching to a saved pair at 216, the robot may, at218, correlate the social action to a mental state indicator. The samemental state indicator may be correlated to different stimulus-responsepairs. Conversely, the same stimulus-response pair may sometimes becorrelated to more than one mental state indicator, but in many caseswill correlate to only one indicator. The mental state indicator may bea symbol (e.g., a binary number) that correlates in any useful way to aset of one or more responses that the robot is capable of performing inreaction to a user's mental state.

At 220, the robot determines whether or not it has identified a mentalstate indicator. If not, it may continue to perform social actions andevaluate user responses at 212. If so, the robot may select one ofsuitable responses associated with the mental state indicator at 222. At224, the robot may detect (e.g., by scanning) available output nodes forperforming the response. At 226, the robot generates a data signal foran available output node, configured to initiate performance of theresponse by the output node. At 230, the robot may transmit the datasignal to the selected output node, causing the output node to perform aresponse as described herein. In performing the response, the outputnode may retrieve a digital copy of video or audio content to be output,either from the robot or from another network node. At 232, the robotmay terminate the session if finished or continue social monitoring at212, if not finished.

FIG. 3 illustrates aspects of a process 300 by a social robot forrecognizing a social action, including the aspect that characterizationof a robot stimulus or user response can be based on different inputdata from independent sensor systems. The robot uses sensor data togeneral various distinctive symbols (e.g., binary numbers) that areassociated with particular user responses. For example, recognition of auser response may include a facial recognition process 302 by the robot,based on camera input data. Recognition of a user response may furtherinclude, for example, a biometric detection process 304, such asdetermining a direction of user gaze, a user pulse, or a user skintemperature. Recognition of a user response may further include, forexample, speech recognition 306 by the robot analyzing audio datacollected by a microphone. Recognition of a user response may furtherinclude, for example, recognizing an environmental circumstance orcontext 308, such as by analyzing photographic data, receiving data froma location beacon or Global Positioning System (GPS) or IoT MEMS,referring to a time-of-day clock, or in any other suitable fashion.Recognition of a user response may further include, for example,determining a relative position and orientation 310 of the robot and theuser, such as by using photographic data and proximity sensor data.Other types of user response parameters may also be determined by therobot, as part of social action recognition, if useful.

At 312, the robot may record current recognition in a cache memory 314and may purge data that is timed out or expired. Similarly, the robotmay maintain symbols indicating robot output stimuli in the cache 314,correlated chronologically to detected user responses. At 316, thesocial robot may filter cached data in the short-term memory 314 using asocial action filter. A social action filter may include, for example, adata structure that links stimulus-response pairs recorded in the datastructure to respective mental state indicators. At 318, if the robotdetects a match, it may at 320 output a mental state indicator. Themental state indicator may be a symbolic (e.g., binary) value that has aspecific correlation (e.g., one-to-one or one-to-many) to a set ofsocial action data, as determined by the social action filter. It shouldbe apparent that the phrase “mental state” in “mental state indicator”is merely a term of convenience and does not limit the manner in whichthe indicator is derived or used. As described, the indicator is amachine-determined symbolic value based on social interactions andcontext known to, or detected by, the robot, and need not correlate tothe user's actual mental state. However, in a well-designedimplementation, the mental state indicator may, and generally should,correspond to a user's mental state insofar as that mental state relatesto social actions that the user is likely to be receptive to. Thedescribed processes for generating a mental state indicator andselecting a social response exemplify processes for emulating humanempathy as expressed by social actions. At 322, the robot may terminatethe session if finished or continue symbol generation at 324, if notfinished.

FIG. 4 illustrates aspects of a social action filter parameter set 400for use in connection with the method 300, organized in a tabular datastructure. The illustrated parameter set 400 includes different symbolicvalues (e.g., binary numbers or character strings) in different datacategories. Each of the values corresponds to a social action withsemantic meaning. For example, a ‘facial expression’ category 402 may bepopulated by one or more values correlated to social actions such asneutral expression, smile, grin, and laughing. An ‘other biometric’category 404 may be populated by one or more values correlated to socialactions such as standing, walking, and pulse in intermediate range. A‘speech’ category 406 may be populated by one or more values correlatedto verbal communication such as laughing, or any of the words funny,riot, hilarious. An environmental circumstances' category 408 may bepopulated by one or more values correlated to social circumstances suchas morning, workday, bright, or at home. A ‘robot-human relationalstatus’ category 410 may be populated by one or more values correlatedto relative positions such as closer than some distance ‘x’, or facingeach other. The illustrated categories and values are merely examples toillustrate underlying concepts, without limiting the method 300 to theseexamples. The parameter set 400 for the social action filter may includeBoolean logic between different symbolic values, for example, a logical“AND” requiring at least one value from any of the categories 402, 404,406, 408 or 410 to be present and a logical “OR” between any values inthe same category, thus finding a match if any one or more values fromeach category is present. Other logical arrangements may also besuitable.

A robot may apply many distinct social action filter parameter sets tofind one or more mental state indicators. In an aspect, the filterparameter sets may be configured such that some combinations of socialactions will correlate to different mental state indicators. When therobot discovers more than one matching mental state indicator, it mayselect one of the indicators for governing a response using any suitableprocess, for example, random selection, quasi-random selection, weightedorder based on past selections (e.g., “taking turns”), or basing theselection on further information (e.g., asking the user, or referring toadditional information such as user preferences, trending popularity, ora robot ‘mood’). In an alternative, if multiple mental states areindicated, the robot may elect to defer its response.

FIG. 5 shows aspects of a method 500 by a social robot for selecting andgenerating an environmental control output based on symbolic outputindicating a mental state from a process such as, for example, themethod 300. At 502, the robot receives or derives a metal stateindicator, in the form of a symbol compatible with the robot'sprocessing system, for example a binary number that falls within a rangeof values that are reserved by the robot's response selection algorithmsfor selecting from available response options. At 504, the robot filtersa current response list 518 for the user, based on comparing the mentalstate indicator with semantic metadata that is independently associatedwith each entry in the response list. The current response list 518 maybe built and maintained by a background process 516 that operates on therobot and/or other nodes of the network that are in communication withthe robot. The list 518 may include every possible response that therobot is capable of, excluding responses that are barred by security,safety, or user preference settings. In an alternative, the list 518 mayinclude a lesser subset of the full set, for example for reducingbandwidth or data storage requirements, for ensuring that freshresponses are used, or for creating a chronological association withcertain responses.

Before continuing with description of FIG. 5, consider how the socialrobot makes use of semantic metadata. FIG. 6 illustrates a response listfor enabling use of semantic metadata 600 for selecting digital contentbased on a mental state indicator. It should be appreciated that thesemantic metadata 600 is expressed at the machine level as amachine-readable symbolic value, for example, as a binary value in arange reserved for expressing semantic concepts. The machine-readablesymbol is one-to-one associated with a human-readable symbol (e.g., aword or phrase as illustrated in FIG. 6) via a database or other datastructure, for example for use in an administrative interface, or in anapplication for crowd-sourcing assignment of semantic metadata toresponse actions. For illustrative simplicity, the table shown in FIG. 6has only two records, a first record in which a response 602 (playLooney Tunes Sylvester the Cat saying “suffering succotash!”) isassociated with semantic metadata 604 (corresponding to thehuman-readable symbols “slapstick” and “surprise-offbeat”), and a secondrecord in which a response 606 (play The Blind Side where Sandra Bullocksays “You threaten my son, you threaten me”) is associated with semanticmetadata 608 (corresponding to the human-readable symbols “protective”and “determined”). It should be appreciated that a response list willoften include many more entries than two, and that response entries inthe table may include other environmental control actions besides “playcontent ‘X’”, for example: “open the blinds,” “turn down the lights,”“brew a cup of chai,” and so forth.

A mental state indicator 610 (with corresponding human-readable symbol“anxious”) is correlated to the semantic metadata 608 as indicated bythe dashed lines and is not correlated to the semantic metadata 604.Accordingly, in the filtering process 504, a robot may exclude the firstresponse 602 and include the second response 606 in the filteredempathetic responses subset 520. Correlations between semantic metadata600 and mental state indicators (e.g., indicator 610) may be recorded invarious ways. For example, the semantic metadata 600 may be expressed asa set of numeric values, in which each numeric value indicates a valuein a range between bipolar opposites for a particular mental stateindicator. Suppose, for example, that a robot makes use of 100 differentmental state indicators, then each response action would be associatedwith 100 or less different values each consisting of the numeric valueand an index number for the mental state indicator (or the mental stateindicator itself) to which the score relates. Continuing the example, arange from −31 to +31 may be indicated by a six-digit binary number,wherein −31 indicates the strongest possible negative correlation, +31indicates the strongest possible positive correlation, and zeroindicates no correlation (e.g., irrelevant or not applicable). Eachresponse action in this example would be scored against one or more ofthe mental state indicators, with unscored indicators assigned a defaultvalue of zero, using a human administrator and/or crowd-sourced data orother metric. Use of numeric score enables the robot to rank responsesby score, and thereby increase the probability of selecting a responsethat is most relevant to any given mental state indicator.

In addition, bipolar scores for semantic relevance may be scored indifferent dimensions (or axes) for different response strategies. Forexample, one strategy may be ‘to express sympathy and support.’ Anothermight be ‘to present an alternative point of view’ (e.g., to argue);still another might be ‘to inject humor,’ as so forth. Each response maybe assigned a separate score based on the applicable strategic axis. Forexample, given the mental state 610 relating to anxiety, the response606 might score strongly positive for ‘sympathy and support’ and neutralor zero for ‘argue’ or ‘make a joke.’ Development of such scores isnecessarily subjective and sensitive to cultural factors, and so may bea good candidate for crowd sourcing, such as by distributing andcollecting data via user surveys or using focus groups.

Referring again to FIG. 5, given the subset 520 of empathetic responses,the robot may apply a second filter 506 based on available outputdevices. For example, some player devices or appliances needed tocomplete a response may not be available or appropriate (e.g., based onuser settings) at the time the response is needed. Therefore, the robotmay eliminate responses that require use of an inactive output device,deriving an actual or constructive second subset 522 that includes onlyavailable empathetic responses.

At 508, the robot may select a response strategy, provided that thesemantic data associated with the responses in the subset 522 enabledifferentiating between the relevance of particular responses fordifferent response strategies, given any particular mental stateindicator. Examples of various response strategies and an associatedscoring system have been provided in the paragraphs above. At themachine level, each response strategy may be implemented as a branch orvariable that causes the selection of response to be based exclusivelyor predominantly on a subset of the semantic metadata that relates tothe response strategy selected by the robot. The selection 508 ofresponse strategy may be based on random or quasi-random factors, suchas a robot mood, or may be based on more deterministic factors such as astrength or clarity score associated with the mental state indicator,recent past mental state indicators in present the user session, userpreference settings, or other factors. Once the robot selects astrategy, the available empathetic responses 522 may be filtered orranked based on the semantic metadata for the selected strategy. Thisderives an actual or constructive third subset 524 that includes onlystrategic available empathetic responses.

At 510, the robot selects one or more of the remaining responses in thesubset 524. Selection from the subset 524 may be done using random orquasi-random factors, so that robot behavior is not deterministic whilestill remaining strategic and empathetic. In an alternative, or inaddition, the selection may be done using a deterministic approach, suchas by choosing the highest entry in a list ranked by a semantic valuescore. The robot may select different responses for different outputdevices, such as, for example, dimming the room lights via a homecontrol unit and playing a romantic clip on a home entertainment centeror vehicle sound system or entertainment system. At 512, the robotencodes instructions for causing the selected response in a data signal,for each output device involved in creating a selected response. Therobot then transmits the data signals to the computer network fordelivery to each involved device, either directly or through anintervening service node (e.g., server 134). At 514, the robot mayterminate the session if finished or continue to the next mental stateindicator at 502, if not finished.

As described in more detail in the '335 Application, social behavior maybe characterized by stimulus-response symbol pairs. FIG. 7A provides anexample of multiple parameters that define a stimulus, and a system 700for organizing the stimulus parameters in logical categories arranged ingroups. Organization of stimulus parameters may be useful forassociating semantic meaning to stimulus-response pairs, and hence, toselecting mental state indicators based on such pairs. While the '335Application taught numeric methods for deriving a personality score, thepresent disclosure points out how the same or similar methods may beused to obtain a mental state indicator for use in the processesoutlined above.

A top-level category 702 defines the stimulus at the highest usefullevel of generality. In the illustrated example, the stimulus is arelative position between the robot or character and something else. Theremaining parameters can be organized in any number of sublevels. Herefor example, the parameters are organized into four groups: a time(when) group 710, an identity (who/what) group 720, a location (where)group 730, and a manner of performance (how) group 740. Each of thegroups contains a score between bipolar opposites of a sort. Forexample, the time group 710 may contain a time-of-day score 712 and aday-of-week score 714, indicating when the stimulus occurs. Each ofthese scores may be normalized to a numeric score that runs continuouslybetween the endpoints of the chosen scale. For example, if an eight-bitscale is used for a time-of-day parameter 712, each 24-hour period fromjust past midnight until midnight of the following day may be dividedinto 128 equal incremented, enabling timekeeping to a granularity of128/24 (5⅓ units per hour, or 11.25 minutes. This should be more thanenough granularity for correlating social responses. Using the sameeight-bit scale to denote time over days of the week results in 7 timesless granularity, or 78.75 minutes, while enabling mood or otherresponse correlations to days of the week. For further examples, anidentity group 720 may include values measuring characteristics of theperson or thing that is positioned to cause the stimulus, for example, aperson/thing scale 720, a gender scale 724, and an extroversion scale726. Likewise, a location group 730 relates where the stimulus occurs,for example an enclosure parameter 732 and a neighborhood characterparameter 734. For further example, a manner of performance parametergroup 740 may include a proximity measure 742, one or more orientationmeasures 744 and a relative motion measure 746.

The system 700 is simplified for illustrative purposes, and the specificmeasures included in the parameter examples are less important than theprinciples of organization and quantification that have been described.Under the illustrated principles, any given stimulus can be representedas a matrix of measurement values. For example, each group can provide arow of the stimulus matrix and each column may delineate betweenadjacent parameters, or vice versa; the matrix may have any usefulnumber of dimensions. A matrix with one or two dimensions is believeduseful; higher dimensions increase complexity but may also be useful.Each cell of the matrix may hold an eight-bit (or other) numeric value,the meaning of which depends on the system 700. It should be appreciatedthat using a standard system for different stimuli would enable rapidmachine comparisons and correlations between different stimuli and theirassociated responses. The particular meaning associated with each cellis not critical, but the value of the cell should have a material impacton the response in at least some cases, or else the parameter is notworth tracking. Accordingly, a general stimulus organization system 700may be organized to enable use of a standard matrix for all stimuli ofinterest, while maximizing the impact of each cell value on theassociated social response.

FIG. 7B shows the same organizing and quantification principles appliedto system 750 for characterization of a social response. In thisexample, the top-level category 752 pertains to speech. The time group760 does not pertain to time of day, etc., because that aspect iscontrolled by when the stimulus occurs. The main time parameter 762pertains to how quickly the speaking response is made after the stimulusis received. The identity group 770 may contain numeric measures of thespeech content, such as the length of speech 772, its humorous character774, its truth 776, and so forth. The actual words spoken cannot readilybe represented numerically in the organization scheme illustrated byFIGS. 7A-B. Instead, for a character that uses distinctive stockphrases, each stock phrase can be scored based on the combination ofcontent identity parameters in the group 720, and thereby identified byscore comparison when socially appropriate for the personality beingexhibited. The position group 780 may contain parameters 782, 784 thatrelate the direction to which the speech is addressed relative to theperson or thing being addressed, or that relate to turning away from theperson or thing being addressed if an oblique response directed away(e.g. turning away and shaking its head) is deemed more appropriate. Amanner of performance parameter group 790 may include a pacing parameter790, a volume parameter 794, and a subtlety parameter 796. Similarorganizational principles may be used for stimulus systems (e.g., 700)and response systems (750), but in both cases care should be taken thatthe measurement parameter for a stimulus does not more truly relate tothe response, and vice-versa.

FIG. 8 illustrates aspects of a flat data structure 800 for holdingquantified multi-parameter data for stimulus or a response. A header 801may include an identifier 802 for a top-level stimulus or responsecategory. The header 801 may further include an indication 806 of thetotal number (‘N’) 808 of subclasses used in the data structure 800.Each subclass identifier 810, 814, 818 may correlate to a parametergroup as described above. The header 801 may be followed by a list ofvalues 812, 816, 820 that may each define several distinct parametersthat may be distinguished by bit position. Parameters within the groupmay be specified by a predetermined number of bits in the numeric values812, 816, 820 in a predetermined order. The data structure 800 comprisesa string of digital values with sufficient index information to make useof the string.

FIG. 9 illustrates graphically a system 900 in which a 4×6 numericstimulus value 902 is paired to a 4×4 numeric response value 904. Inaddition, the illustrated stimulus-response pair 902, 904 correlates toa single 1×8 numeric mental state indicator 906 for the pair, based onany useful scale. In the illustrated example, black indicates a maximumvalue, white represents a minimum value, gray represents someintermediate value, and “X” represents a null (inapplicable) value. Eachcell or box represents a parameter value for a stimulus or response.Hence, the system 900 illustrates how a particular mental stateindicator 906 can be correlated to one or more numerically representedstimulus-response pairs, using purely numerical methods.

For example, given a large set of many stimulus-response pairs allnumerically represented under a consistent scheme, a processor maydetermine a correlation coefficient between each cell of the stimulusmatrix and each cell of the response matrix. For each cell of thestimulus matrix, the processor may determine the most positivelycorrelated response parameter and the most negatively correlatedresponse parameter, over the entire behavior set, using a measure suchas, for example, a Pearson's correlation coefficient. For example, thevalue of the stimulus parameter at row-column (1, 1) may correlate mostpositively with the response parameter at (1, 4) and most negativelywith the response parameter at (2, 2). If so, the first two values ofthe mental state indicator 906 for the stimulus response pair 902, 904as a whole may indicate the value of the correlation. For example, theamount of positive correlation (e.g., a number between 0 and 1) may beindicated by seven binary digits plus one digit to indicate the sign(positive or negative), while the amount of negative correlation can beindicated using the same scheme. If this is repeated for every stimulusvalue, a numeric mental state indicator for the stimulus-response paircan be determined without any need for a human to assessstimulus-response pairs qualitatively. This may be done for everydistinct set of stimulus-response pairs identified with a particularuser, to provide a custom set of mental state indicators for the user.In an alternative, or in addition, stimulus-response data from differentusers may be aggregated to provide aggregate mental state indicators.The meaning of a purely numerically defined mental state indicator will,unlike a predetermined standardized scale, depend on thestimulus-response from which it is derived and the numeric method usedto derive it. So long as the numeric method reflects a rational measureof correlation between stimuli and responses for the behavior set, anumerically derived mental state indicator should work just as well ifnot better than one derived by more qualitative methods.

A quantitative mental state indicator that is defined using a purelyautomated, numeric process may not have any meaning that can easily beexpressed in words, unlike the examples described herein above. Theautomatically determined indicator nonetheless enables a social robot toselect appropriate responses, because the indicator will capture themost important correlations between stimuli and responses. Consider, forexample, how a numerically determined mental state indicator derivedfrom a particular stimulus-response pair (902, 904) works in associatingwith semantic metadata as described in connection with FIG. 6. Themental state indicator will need to be scored relative to semanticmetadata as before. However, scores can be interpolated between selectedkey mental state indicators. For example, if a particular mental stateindicator is believed to indicate an ‘anxious’ mental state particularlywell, it may be designated as a ‘key’ value for an anxious mental state.Then, semantic association scores for any similar mental state indicatorcan be interpolated between the indicator's closest key value and itsnearest key neighbors.

For multi-factor mental state indicators, different factors may be moreor less relevant to different response strategies, because each factorrelates to a particular stimulus/response factor. Some stimulus/responsefactors are more relevant to certain response strategies than to others.For example, a ‘mirroring’ factor, measuring how well the user ismirroring the social robot's actions, may be positively correlated to astrategy of expressing sympathy and negatively correlated to a strategyof providing a contrasting opinion. The semantic association scores cantherefore also be interpolated between different key mental stateindicator values for different strategies, as previously described.Advantageously, use of a numerically determined measure for a mentalstate indicator may facilitate numeric determination of associations tosemantic metadata, reducing the need for human involvement.

In general, whether or not a mental state indicator is derived purelynumerically or is based on some human-comprehensible symbol, scoring anyparticular stimulus-response pair based on the applicable scale shouldenable a social robot to later identify a proper subset of all of therobot's social behaviors containing only those behaviors within adesired degree of similarity (e.g., plus or minus 10%) of a specificpersonality profile for the current character and mood, or that areflagged as being part of the behavior set from which the currentpersonality profile was developed. Regardless of how the personalityprofile is developed, operation of the social robot in real time isalways fully automated based on algorithms programmed into the robot'scontrol processor(s).

An example of a process 1000 for response characterization is diagrammedin FIG. 10, for performance by a processor of a social robot. Responsecharacterization facilitates the creation of stimulus-response pairsfrom for mental state indication. At block 1002, the processor receivessensor data (if a real social robot) or modeled environmental data (if avirtual social robot) via a data stream or file, and records thereceived data in a cache memory 1004. At 1006, the processor analyzesdata from the cache and detects one or more user events, using anysuitable event detection method as known in the robotic arts. The eventsdetected are social events generated by the user, for example, smiling,turning to face the robot, turning away from the robot and muttering,and so forth. The processor places identifiers for the detected eventsin an events cache 1008. In parallel to the event detection 1006, theprocessor compares 1010 contents of the current event cache 1008 to therobot's user response library 1012 in a computer memory, which containsall user responses associated with any defined social stimuli output bythe robot. If no response is detected 1016, the process loops back toblock 1002 until operation of the social robot is finished at 1020. If aresponse is detected at 1016, the processor signals an alert 1014 thatindicates an identifier for the detected response for use in building acurrent stimulus-response pair. Ignoring the robot may be a socialresponse or may be unintended. Therefore, the lack of a response may beidentified as an ‘intentionally ignoring’ response if the stimulus andenvironmental conditions are such that it is likely to be intentional.The robot is itself responsible for producing the related stimulus, somay respond to the alert by recording the occurrence of a particularstimulus-response event. Once the alert is provided at 1014, theprocessor flushes 1018 the caches 1004 and 1008, and the process 1000loops back to block 1002 until operation of the social robot isfinished.

Each of the foregoing methods may be implemented as modules in, orprocesses by, a real social robot 1100 for example as shownschematically in FIG. 11. The social robot 1100 may includeelectro-mechanical components including an electrical circuit configuredas a computer housed in an exoskeleton or housing 1130. The circuit mayinclude a central processing unit or units 1102, operatively coupled tomotor driver controller 1120 driving a motor 1122, which moves a loadcomprising a mechanism connected to a head or body portion of the socialrobot 1100, causing the head or body portion to move in a sociallysignificant way. The processor determines responses and provides motioncontrol signals to the motor driver 1120, based on a behavioral responseselected as described herein above. The motion control signals cause themotor 1122 to move the connected body part in conformance with theresponse selected by the processor 1102.

The robot 1100 may contain other components for computing in addition tothe processor 1102. For example, the processor circuit may be coupled toa user interface component 1104, such as for example a keypad ortouchscreen, for entry of user controls. The processor circuit may becoupled to a display component 1110, for example a touchscreen or otherdisplay. The display 1110 may be used for a user interface, to displayexpressive features of the social robot (e.g., a face), or for both. Theprocessor circuit may be coupled to one or more sensors 1106 configuredfor sensing one or more environmental conditions of the environmentaround the robots, or of objects that the robot interacts with. The oneor more sensors 1106 may include, for example, a camera, light sensor,microphone, temperature sensor, pressure sensor, accelerometer, touchsensor, proximity sensor, pulse sensor, gaze direction sensor, or othersensing or IoT MEMS device. The processor circuit may be coupled to acommunications port 1108 for coupling to a computer network, or to adata communication network for receiving data and program instructions.The processor circuit may further be coupled to an audio outputtransducer 1118 for outputting robot vocalizations. The processorcircuit may further be coupled to a volatile random access memory device1114 and to a non-volatile memory 1116 for long term storage of programinstructions and data.

An example of a real social robot 1200 is shown in isometric view inFIG. 12. The social robot 1200 comprises an assembly of a smartphone1202 (or similar computing and wireless communication device) with amechanical base. The smartphone 1202 couples via a wired or wirelessinterface to a power and control circuit in the base 1204. Separately ortogether, the smartphone 1202 and base unit 1204 include an electronicsubstrate holding one or more processors coupled to a memory and to atleast one circuit for driving components of the robot. A circuit in thebase unit may be similar to the circuit described in connection withFIG. 11, minus a touchscreen display which is supplied by thesmartphone. The circuit in the smartphone may also be similar to thecircuit of FIG. 11, minus the motor and motion components. Together, thebase unit and smartphone comprise a circuit with all of the componentsdepicted in FIG. 11. For example, the robot's sensors may include acamera 1226 mounted in the body of the smartphone 1202.

In some embodiments, the main processor and application for controllingoperation of the social robot is located in the smartphone 1202, with aprocessor in the base unit 1204 operating as a slave or client system.In alternative embodiments, the main processor and application forcontrolling operation of the social robot is located in the base unit1204, with the smartphone 1202 playing a client or slave role. The dualdevice, modular construction provides an advantage of enabling acommonly available, versatile device (the smartphone) to supply manynecessary components of the robot 1200, while still allowing it to beused as a smartphone at other times. Nonetheless, the methods herein forcontrolling real social robots apply equally well to modular and unitarydesigns for robots.

In the robot 1200, a face component comprises the display screen 1216,displaying human-recognizable expressive features 1218 controlled by thesmartphone processor, including a configurable brow 1220, a configurablepair of eyes 1222, and a configurable mouth 1224. These components maybe animated based on a personality profile for a character to beemulated. The screen 1216 and smartphone audio transducer may also beused to output an environmental response, for example playing a selectedaudio-video clip. The robot 1200 also includes wireless communicationfeatures through which it may send data signals for controllingenvironmental nodes in response to the user's social behavior, asdescribed in more detail herein above.

The robot 1200 further includes a movable head component coupled to theface component. The moveable head component includes the smartphone1202. While the robot 1200 lacks a humanoid torso or appendages, thesemay be added if desired. The movable head (phone 1202) itself may becontrolled at least in part by a processor in the smartphone 1202, whilemovement of the base unit 1204 and its included rotating ‘neck’component 1206 may be controlled at least in part by a processor in thebase unit 1204, that is in communication with the processor in thesmartphone 1202.

The robot 1200 may further include an audio output transducer (notshown) coupled to a control circuit of the smartphone 1202. In thealternative, or in addition, an audio output transducer may be locatedin the base unit 1204. Either or both audio output transducers may beused for the robot's social vocalizations. An audio transducer may alsobe used as a component of a proximity sensing sonar system.

The base housing 1208 may be configured as a torso component coupled tothe head component (smartphone 1202) via a neck mechanism 1206controlled by one or more processors in the base unit 1204. The neckmechanism 1206 permits rotation of the head component 1202 relative tothe torso component 1208. The torso component (housing) 1208 mayconnected to a control circuit via motor and motor drivers locatedinside the base housing 1208. The base housing 1208 may be supported bywheels or casters (one shown at 1210), at least one of which is drivenby a motor under control of the robot's central processor, and at leastone of which can be steered. Accordingly, the housing 1208 can move androtate over any navigable hard surface. The ‘head’ 1202 can also turn toface any user, person, or object in the area, being provided with twodegrees of rotational freedom by the neck and torso components 1206,1208.

To prevent collisions or falls, a pair of proximity sensors 1212, 1214and/or the camera 1226 may provide navigation data to a navigationalengine executing in a processor of the robot. The navigational enginemay limit or alter movement of the movable torso 1208 and neck 1206 toprevent collisions and falls, while otherwise providing output motion inconformance with requested social behaviors.

To that end, a computer memory of the robot 1200 may hold programinstructions, that when executed by its one or more processors, causesthe social robot to provide an empathic environmental response (e.g.,playing a video clip) to the user's social interactions with the robot,and to perform more detailed operations as described herein.

The empathic environmental control techniques described herein may beapplied equally well to virtual social robots, which can interactsocially with a user in a virtual environment and communicate withnetwork nodes in the user's real or virtual environment. Hence, avirtual social robot may implement any of the detailed operationsdescribed herein that do not require actions by a physical robot.

For example, FIG. 13 shows hardware components of a virtual social robot1300, in block diagram form. The virtual social robot 1300 includes acontrol processor 1302, that may be used to generate the virtualenvironment and the virtual social robot in it, or that may be dedicatedexclusively to the social robot or to the social robot and some limitedpart of its virtual environment. The processor 1302 may be coupled to ashort-term memory 1304 (e.g., RAM) enabling execution of programinstructions, and a non-volatile data store 1306 for long-term storageof virtual data and program instructions. The processor 1302 may becoupled to a communications port for sending and receiving data via adata communications or computer network.

The virtual social robot 1300 may be expressed via at least two outputs:audio and video. The social robot 1300 may include an audio drivercomponent 1312 coupled to the processor 1302, producing an audio signal1313 for providing to an audio output transducer 1320. The social robot1300 may also include graphics driver component 1310 coupled to theprocessor 1302, producing a video signal 1313 for providing to a videodisplay device 1318. The virtual social robot 1300 may hold programinstructions in the memory 1304, that when executed by the processor1302, causes the social robot to provide an empathic environmentalresponse (e.g., playing a video clip) to the user's social interactionswith the robot, and to perform more detailed operations as describedherein.

FIG. 14 shows an example of hardware or firmware components 1400 of avirtual social robot. Hardware components of the robot may be consistentwith those shown and described in connection with FIG. 13. The softwareof firmware components 1400 may include a geometric mesh 1402, anarmature or system of bones 1404, a set of mesh deformation rules 1406governing how the mesh responds to movement of the armature 1404, andsurface rendering data 1408 (e.g., diffuse, specular and bump orelevation maps). The software or firmware components may include anysuitable elements as known in the art of model manipulation andrendering. The components may be configured in any desired way toresult, when rendered, in a virtual social robot of any desiredappearance 1410. A single virtual robot having a certain personality mayeven take on different appearances as a form of social expression.Technical aspects of manipulating and rendering virtual characters arewell understood in the art and may readily be adapted to make use ofenvironmental control based on social interaction techniques asdescribed herein for social robots generally.

Referring to FIG. 15, a real social robot and a virtual social robot maybe coordinated 1500 to represent a unitary real-virtual social robot.For example, a real mobile social robot carrying a locating device mightmove around a physical play space for a virtual reality game (e.g., bymoving over the floor, flying through the air, climbing walls, orcombinations of these or other motions). One or more users each wearingvirtual reality (or augmented reality) gear may see and hear a virtualsocial robot rendered in a virtual environment (or augmenting a realenvironment) at a location corresponding to a current location of thereal social robot. The real social robot may move in response to theuser actions in the real environment and in the virtual environment. Forexample, a game of Quidditch might be played in this way, or a similargame in which the Quidditch snitch is replaced by a character withpersonality, such as, for further example, a game of ‘capture BugsBunny.’ Many other use cases for mixed social robots are also possible.In each of these use cases, a control processor for the mixed realitysession may allocate a unitary (total) behavior response set 1502 to areal response set 1504 and to a virtual social robots response set 1506.For example, overall motion response may be allocated to the real robot,and other responses (e.g., facial expressions and speech) allocated tothe virtual robot. Stimuli is for triggering the response may,conversely, be detected by aggregating stimuli detected in the real andvirtual contexts. The control processor may further synchronize theresponse sets 1504, 1506 so that the responses of the real social robotand the virtual social robot are coordinated. In other respects,providing an empathic environmental output (e.g., playing a video clip)in response to the user's social interactions with the robot may beperformed as described elsewhere herein.

In view the foregoing, and by way of additional example, FIG. 16 showsaspects of a method 800 or methods for control of a social robotproviding an empathic environmental output (e.g., playing a video clipor other non-robotic action) in response to the user's socialinteractions with the robot. The method 1800 may be performed by acomputer processor controlling a real social robot, a mixed realityoutput device, a computer network, or combination of the foregoing, andcomponents connected to the processor as described herein above. A realor virtual social robot that performs the method 1600 may include one ormore processors coupled to at least one circuit for driving componentsof the social robot, such as, for example, the processor 1102 andassociated circuit shown in FIG. 11. The social robot may furtherinclude a social expression component connected to the at least onecircuit, the social expression component comprising a human-recognizableexpressive feature controlled by the one or more processors according toa social behavior algorithm. For example, a real social robot mayinclude a display screen 1216 portraying a face as shown in FIG. 12,while a virtual social robot may include rendering and modelingcomponents for a character 1410 with a face. The social robot performingthe method 1600 may further include at least one of a camera connectedto the at least once circuit, positioned for observing a userinteracting with the social robot or a microphone connected to the atleast one circuit, positioned for receiving sound from the user. Forexample, the real social robot 102 shown in FIG. 1 includes both acamera 114 and a microphone 116. Virtual social robots may use a headsetinterface and receive data from a headset-mounted camera or microphonepositioned to sense user actions. The social robot performing the method1600 may further include a program memory, coupled to the at least onecircuit and to the one or more processors, the program memory holdingprogram instructions configured for execution by the one or moreprocessors, causing the social robot to perform operations of themethod. For example, a social robot may include a memory 106 as shown inFIG. 1, RAM 114 as shown in FIG. 11, or memory 1304 as shown in FIG. 13.

The method 1600 may include, at 1610, interacting socially with the userby operating a social expression component. The social expressioncomponent may be coupled to the at least one circuit for drivingcomponents of the social robot. The social expression component mayinclude a human-recognizable expressive feature (e.g., eyes, brow, mouthdisplayed on a video screen) controlled by the at least one circuitaccording to a social behavior algorithm. The social behavior algorithmmay select social actions (e.g., smiling, frowning, speaking, turning,moving) for the robot, based on a detected social setting and usersocial responses.

The method 1600 may further include, at 1620, determining, by the one ormore processors, user behavior based at least in part on data from avideo camera and a microphone coupled to the at least one circuit.“Determining” is meant in the sense of recognizing the user behavior,based on input from a video camera or microphone. In an aspect, thesource of information for the determining 1620 may include one or morephysical sensors coupled to the processor, and the operation ofdetermining 1620 includes recognizing the set of events based on datafrom the one or more sensors. Social action by the user excludes theissuing of a command by the user to the robot, with the intent that therobot automatically carry out the user command. However, social actionmay include a request by the user, in a context where the robotdetermines its response at least partly based on a social content inwhich the request is made, and not solely on the content of the request.The determining 1620 may include various modes of social actionrecognition as described, for example, in connection with blocks 302-310of FIG. 3. Other techniques for social action recognition may also beused.

The method 1600 may further include, at 1630, correlating, by the one ormore processors, the user behavior to social behavior of the socialrobot. The correlating 1630 may include filtering a cache of action dataas described, for example, in connection with block 316 of FIG. 3 andFIG. 4, or a purely numeric correlation as described in connection withFIGS. 9-10. As explained herein above, the correlating develops a mentalstate indicator, and it is the mental state indicator that enablesselection of empathic environmental action in response to social conductof the user. Various examples of empathic action have be provided in theforegoing disclosure, for example, playing a selected video clip,altering the room lighting, playing mood music, and so forth.

The method 1600 may further include, at 1640, generating a data signalbased on the correlating. For example, the processor may, afterselecting a response for an output device in the user's environment,encode instructions for causing the output device to perform theresponse on a communications layer of a network to which the outputdevice is connected. The method 1600 may further include, at 1650,outputting the data signal configured for causing an electronic outputdevice to respond to information encoded by the data signal. Forexample, the processor may send the data signal to a transmittercomponent of the social robot, which transmits it to the output device.

In an embodiment of the method, the robot includes an electronicsubstrate holding one or more processors coupled to a memory and to atleast one circuit for driving components of the robot. In suchembodiments causing the robot to perform the response may includesending an electrical signal to the circuit, wherein the signal isconfigured to cause a motor connected to the circuit to move a part ofthe robot (e.g., a head, neck, or appendage). In addition, or in analternative, the robot includes an electronic substrate holding theprocessor operatively coupled to a memory and to a social expressioncomponent, the social expression component comprising ahuman-recognizable expressive feature controlled by the one or moreprocessors according to a social behavior algorithm. In suchembodiments, causing the robot to perform the response may includesending an electrical signal to the social expression component, thesignal configured to cause movement of the expressive features. Forexample, the social expression component may include a video displayscreen, and the robot may send a video signal that encodes one or morehuman-recognizable expressive features (for example, at least two suchfeatures) to the display screen, causing animation of those features ina facial expression.

In an alternative embodiment, the robot includes a processor coupled toa memory, to a user input port, and to a video output device, forexample, a video screen, projector, virtual reality headset, oraugmented reality headset. The user experience of the social behavior ofthe robot may be entirely virtual. In such embodiments causing the robotto perform the response may include modeling an animation of a model ina virtual computer environment, rendering the environment and robot, andsending an electrical signal to the video output device, causing it tooutput a video or equivalent moving image of the robot performing theresponse. In such embodiments, the source of information about eventsexperienced by the robot may include a computer model of the virtualenvironment in which the robot acts and recognizing the stimulus mayinclude tracking activity of the computer model.

FIGS. 17 and 18 show additional operations 1700 and 1800 that may beincorporated into the method 1600. Any one or more of the additionaloperations 1700 or 1800 may be combined, in any operative order, withother operations of the method 1600. Organization of the operations 1700and 1800 into separate figures is merely for illustrative convenienceand should not be interpreted as significant to the disclosed method.

Referring to FIG. 17, the method 1600 may include any one or more of theadditional operations 1700. The method 1600 may include, at 1710,determining a current mental state indicator based on the correlating.The mental state indicator may be determined as described, for example,in connection with block 316 of FIG. 3 and FIG. 4, or as described inconnection with FIGS. 9-10. In an alternative, the correlating may beaccomplished without deriving a single symbolic indicator, such as, forexample, by determining multiple indicators, or by using a relationaldata structure to correlate user behavior to social behavior of therobot.

The method 1600 may include, at 1720, causing selection of an identifierof a prerecorded audio-video content clip based on comparing semanticmetadata associated with the identifier to the current mental stateindicator. More detailed aspects of selecting a pre-recorded audio-videoclip are described herein above in connection with FIGS. 5 and 6. In anaspect, a local or remote database of content may be selectively sharedwith the user of the social robot, in response to the current mentalstate indicator or in response to an anticipated future condition. Forexample, the database of content may include video clips, audio-videoclips, or audio clips that are indexed based on their semantic content.The social robot may be configured to output a selected one of the clipsfor any social purpose. For example, if the social robot senses that itsuser is unhappy, it may select a humorous clip from clips of the user'sknown favorite content or characters. For further example, if the socialrobot is commenting on a sensed social situation, for example, toencourage the user to get some exercise, it might play a clip that ismarked as appropriate for users with personal or demographiccharacteristics that matches the user's that in some way reminds theuser to get up and do some exercise. Accordingly, users may beentertained by the context and selection of clips selected by the socialrobot's content selection algorithm. In more detailed aspects, and forfurther example, the robot may:

learn what type of relevant content clips the current user or users arelikely to pay attention to, for example by identifying areas, genres,works (prose, poetry, pictures, videos, art, etc.) that interest thetarget user(s) by questioning or otherwise ‘testing’ the target users(s)for interest levels, and then audio-visually output the selected contentclips;

wirelessly send selected content clips (or have them sent from a remoteserver) to audiovisual display gear (e.g., speakers, a phone, a tablet,a PC, a smart TV, an AR headset/glasses, etc.) independent of the robotso that the target user(s) can see and/or hear the content clipsynchronized to a social interaction involving the social robot, or inresponse to an authorized user request;

command a wireless device such as, for example, the Amazon™ Echo™ andFire Stick™, to tune in or otherwise obtain relevant content for thetarget user(s), wherein the content is selected by the social robotbased on user/situational parameters or selected by the user.

The method 1600 may include, at 1730, transmitting the current mentalstate indicator to a remote server, and receiving the identifier for thepre-recorded audio-video content clip from the remote server.Accordingly, the robot may rely on another network node to performcontent selection. The selected content may be streamed or otherwiseprovided from any suitable network node to the selected output device.In an aspect, the social robot's memory may hold identifiers forprerecorded audio-video content clips each associated with semanticmetadata pertaining to corresponding ones of the content clips. In suchcase, the method 1600 may include, at 1730, causing the social robot toperform executing a content-selection algorithm, thereby causing theselection of the identifier.

The method 1600 may include, at 1740, performing the correlating atleast in part by determining the current mental state indicator suchthat the indicator indicates at least one of an indicator of a user moodor an indicator of a user desire. For example, a frown may beinterpreted as indicating an unhappy mood. Restless behavior may beinterpreted as indicating a desire for something different orstimulation.

In an aspect, the electronic output device may be, or may include, anaudio-video output device. The method 1600 may include, at 1750, causingthe social robot to perform configuring the data signal so that theprerecorded audio-video content clip is output by the audio-video outputdevice. For example, the social robot may stream the audio-video clip tothe output device, send it over an internal bus to a video output deviceintegrated into the social robot's circuitry, or send a file containingthe audio-video clip to the output device. The audio-video output devicemay be coupled to the social expression component, for example, a videooutput device may be coupled to the social robot over a computernetwork. In an alternative, the social expression component may be, ormay include, the audio-video output device, for example, a video screenfor supplying the robot's facial expressions may also serve foroutputting the video.

Referring to FIG. 18, the method 1600 may include any one or more of theadditional operations 1800. The method 1600 may include, at 1810,configuring the data signal for causing the electronic output device torespond, wherein the electronic output device comprises at least one ofa home appliance or motor vehicle and the information encoded by thedata signal comprises operating instructions for the home appliance ormotor vehicle. Examples of operating a home appliance in response to thecorrelating (e.g., using a mental state indicator) are described above,for example, in connection with FIG. 1.

As noted above, the social robot may include a specialized biometricsensor, for example, a pulse sensor, skin temperature sensor, or gazedirection sensor. The biometric sensor may be coupled to at least onecircuit for providing a data signal responsive to a bodily state of theuser. In such case, the method 1600 may include, at 1820, determiningthe current mental state indicator further based on a data signal fromthe biometric sensor. The biometric sensor may be coupled to the atleast one circuit via an intervening wireless receiver and transmitterpair.

The method 1600 may include, at 1830, selecting a response to be causedby sending the data signal to the output device further based on adetected context in which a social interaction with the user isoccurring. For example, the social robot may select different contentfor play, depending on whether or not the user is a home, in an officeenvironment, or is near other people. In an aspect, the method mayinclude, at 1840, selecting the response further based on a locationbeacon or other IOT MEMS signal. For example, the social robot mayreceive a wireless beacon from a transmitter located in the area, withlocation or current event information. The social robot may include thelocation or current event information as an additional filter thatexcludes otherwise relevant empathic actions. In an alternative, therobot may use the beacon information as an attractor, that includesadditional empathic actions in the response list. The method 1600 mayinclude further, more detailed operations in connection with the stepssummarized above, as described in the foregoing description.

FIG. 19 is a conceptual block diagram illustrating components of anapparatus or system 1900 for control of a social robot providing anempathic environmental output (e.g., playing a video clip or othernon-robotic action) in response to the user's social interactions withthe robot. The apparatus or system 1900 may include additional or moredetailed components for performing functions or process operations asdescribed herein. For example, the processor 1910 and memory 1916 maycontain an instantiation of a process for providing an empathicenvironmental output in response to the user's social interactions withthe robot. As depicted, the apparatus or system 1900 may includefunctional blocks that can represent functions implemented by aprocessor, software, or combination thereof (e.g., firmware).

As illustrated in FIG. 19, the apparatus or system 1900 may comprise anelectrical component 1902 for interacting socially with the user. Thecomponent 1902 may be, or may include, a means for said interacting.Said means may include the processor 1910 coupled to the memory 1916,the processor executing an algorithm based on program instructionsstored in the memory. Such algorithm may include a sequence of moredetailed operations for interacting socially with the user, for example,as described in connection with FIGS. 1-2. For example, said means mayinclude a social expression component coupled to at least one circuitfor driving components of the social robot. The social expressioncomponent may include a human-recognizable expressive feature controlledby at least one circuit of the robot according to a social behavioralgorithm, wherein the at least one circuit comprises one or moreprocessors.

The apparatus 1900 may further include an electrical component 1903 fordetermining user behavior based at least in part on data from a videocamera and a microphone coupled to the at least one circuit. Thecomponent 1903 may be, or may include, a means for said determining.Said means may include the processor 1910 coupled to the memory 1916 andto at least one sensor (not shown), the processor executing an algorithmbased on program instructions stored in the memory. Such algorithm mayinclude a sequence of more detailed operations, for example, asdescribed in connection with FIG. 3 at blocks 302-310.

The apparatus 1900 may further include an electrical component 1904 forcorrelating the user behavior to social behavior of the social robot.The component 1904 may be, or may include, a means for said correlating.Said means may include the processor 1910 coupled to the memory 1916,the processor executing an algorithm based on program instructionsstored in the memory. Such algorithm may include a sequence of moredetailed operations, for example, any of the processes for deriving amental state indicator as described herein above, such as in connectionwith FIG. 4, or FIGS. 9-10.

The apparatus 1900 may further include an electrical component 1905 forgenerating a data signal based on the correlating. The component 1905may be, or may include, a means for said generating. Said means mayinclude the processor 1910 coupled to the memory 1916, the processorexecuting an algorithm based on program instructions stored in thememory. Such algorithm may include a sequence of more detailedoperations, for example, receiving identifiers for a desired action byan output device, and encoding an instruction to perform the actionaccording to an application interface for the output device.

The apparatus 1900 may further include an electrical component 1906 foroutputting the data signal configured for causing an electronic outputdevice to respond to information encoded by the data signal, generatingan empathetic response. The component 1906 may be, or may include, ameans for said outputting. Said means may include the processor 1910operatively coupled to the memory 1916, and to one or more of the motordriver 1912, the audio adapter 1913, the display adapter 1914, or to anetwork interface module, the processor executing an algorithm based onprogram instructions stored in the memory. Such algorithm may include asequence of more detailed operations, for example, addressing the datasignal to a selected output device, providing the addressed data signalto a network interface module, and sending the data signal to the outputdevice via the network interface module.

The apparatus 1900 may optionally include a processor module 1910 havingat least one processor. The processor 1910 may be in operativecommunication with the modules 1902-1906 via a bus 1913 or similarcommunication coupling. In the alternative, one or more of the modulesmay be instantiated as functional modules in a memory of the processor.The processor 1910 may effect initiation and scheduling of the processesor functions performed by electrical components 1902-1906.

In related aspects, the apparatus 1900 may include a network interfacemodule (not shown) operable for communicating with system componentsover a computer network, instead of or in addition to the transceiver1912. A network interface module may be, or may include, for example, anEthernet port or serial port (e.g., a Universal Serial Bus (USB) port).In further related aspects, the apparatus 1900 may optionally include amodule for storing information, such as, for example, a memory device1916. The computer readable medium or the memory module 1916 may beoperatively coupled to the other components of the apparatus 1900 viathe bus 1913 or the like. The memory module 1916 may be adapted to storecomputer readable instructions and data for effecting the processes andbehavior of the modules 1902-1906, and subcomponents thereof, or theprocessor 1910, or the method 1600 and one or more of the additionaloperations 1700 or 1800 disclosed herein. The memory module 1916 mayretain instructions for executing functions associated with the modules1902-1906. While shown as being external to the memory 1916, it is to beunderstood that the modules 1902-1906 can exist within the memory 1916or an on-chip memory of the processor 1910.

The apparatus 1900 may include a transceiver (not shown) configured as awireless transmitter/receiver, or a wired transmitter/receiver, fortransmitting and receiving a communication signal to/from another systemcomponent. In alternative embodiments, the processor 1910 may includenetworked microprocessors from devices operating over a computernetwork. In addition, the apparatus 1900 may include an audio adapter1913 for providing an audio signal to an audio output device, causingthe output device to vocalize a social robot response to a user. Theapparatus 1900 may include display adapter 1914 for providing a videosignal for a virtual environment, a social robot output for a user, orboth. The apparatus 1900 may include a motor driver for providingelectrical power to a motor, causing the motor to move a part of thesocial robot in a manner that provides a social action for the user toexperience.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the aspects disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

As used in this application, the terms “component”, “module”, “system”,and the like are intended to refer to a computer-related entity, eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component or a module may be, but are notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component or a module. One or morecomponents or modules may reside within a process and/or thread ofexecution and a component or module may be localized on one computerand/or distributed between two or more computers.

Various aspects will be presented in terms of systems that may include anumber of components, modules, and the like. It is to be understood andappreciated that the various systems may include additional components,modules, etc. and/or may not include all of the components, modules,etc. discussed in connection with the figures. A combination of theseapproaches may also be used. The various aspects disclosed herein can beperformed on electrical devices including devices that utilize touchscreen display technologies, heads-up user interfaces, wearableinterfaces, and/or mouse-and-keyboard type interfaces. Examples of suchdevices include VR output devices (e.g., VR headsets), AR output devices(e.g., AR headsets or glasses), computers (desktop and mobile), smartphones, personal digital assistants (PDAs), tablets, motor vehicles orvessels, including self-driving cars, and other electronic devices bothwired and wireless.

In addition, the various illustrative logical blocks, modules, andcircuits described in connection with the aspects disclosed herein maybe implemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general-purpose processor may be a microprocessor,but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

Operational aspects disclosed herein may be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, digital versatile disk (DVD),Blu-ray™, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in an ASIC. The ASIC mayreside in a client device or server. In the alternative, the processorand the storage medium may reside as discrete components in a clientdevice or server. In another alternative, operational aspects disclosedherein may be embodied in a stream of software that is fed in real timeor near-real time to the robot hardware that is then executed by aprocessor or software module, or a combination of the two. In thismanner, computational power can be off-loaded to the cloud so that therobot's on-board processing can be limited while the cloud-basedcomputational power can be virtually unlimited, allowing moresophisticated inputs, subsequent analyses, recognition and/or responses,and related computational tasks to be accomplished in the cloud.

Furthermore, the one or more versions may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedaspects. Non-transitory computer readable media can include but are notlimited to magnetic storage devices (e.g., hard disk, floppy disk,magnetic strips, or other format), optical disks (e.g., compact disk(CD), DVD, Blu-ray™ or other format), smart cards, and flash memorydevices (e.g., card, stick, or other format). Those skilled in the artwill recognize many modifications may be made to this configurationwithout departing from the scope of the disclosed aspects.

The previous description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments without departing from the spirit or scopeof the disclosure. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope consistent with the principles and novel features disclosedherein.

In view of the exemplary systems described supra, methodologies that maybe implemented in accordance with the disclosed subject matter have beendescribed with reference to several flow diagrams. While for purposes ofsimplicity of explanation, the methodologies are shown and described asa series of blocks, it is to be understood and appreciated that theclaimed subject matter is not limited by the order of the blocks, assome blocks may occur in different orders and/or concurrently with otherblocks from what is depicted and described herein. Moreover, not allillustrated blocks may be required to implement the methodologiesdescribed herein. Additionally, it should be further appreciated thatthe methodologies disclosed herein are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethodologies to processing circuits for social robots.

The invention claimed is:
 1. A social robot, comprising: one or moreprocessors coupled to at least one circuit for driving components of thesocial robot; a social expression component connected to the at leastone circuit, the social expression component comprising a display screenconfigured for displaying facial expressions controlled by the one ormore processors according to a social behavior algorithm; at least oneof a camera connected to the at least one circuit, positioned forobserving a user interacting with the social robot or a microphoneconnected to the at least one circuit, positioned for receiving soundfrom the user; and a program memory, coupled to the at least one circuitand to the one or more processors, the program memory holding programinstructions configured for execution by the one or more processors,causing the social robot to perform operations comprising: interactingsocially with the user by the social expression component, determininguser behavior based at least in part on data from the at least one ofthe video camera or the microphone, correlating the user behavior tosocial behavior of the social robot, generating a data signal based onthe correlating, comprising determining a current mental state indicatorbased on the correlating, and causing selection of an identifier of aprerecorded audio-video content clip based on comparing semanticmetadata associated with the identifier to the current mental stateindicator, and transmitting the data signal configured for causing anelectronic output device to respond to information encoded by the datasignal.
 2. The social robot of claim 1, wherein the instructions arefurther configured for causing the social robot to perform the causingselection of an identifier at least in part by scoring response actionsagainst the mental state indicator using a numeric scoring system. 3.The social robot of claim 2, wherein the instructions are furtherconfigured for causing the social robot to perform transmitting thecurrent mental state indicator to a remote server and receiving theidentifier from the remote server.
 4. The social robot of claim 2,wherein the memory holds identifiers for prerecorded audio-video contentclips each associated with semantic metadata pertaining to correspondingones of the content clips, and wherein the instructions are furtherconfigured for causing the social robot to perform executing acontent-selection algorithm, thereby causing the selection of theidentifier.
 5. The social robot of claim 2, wherein the correlatingfurther comprises determining the current mental state indicatorcomprising at least one of an indicator of a user mood or an indicatorof a user desire.
 6. The social robot of claim 2, wherein the electronicoutput device comprises an audio-video output device, and theinstructions are further configured for causing the social robot toperform configuring the data signal so that the prerecorded audio-videocontent clip is output by the audio-video output device.
 7. The socialrobot of claim 5, further comprising the audio-video output devicecoupled to the social expression component.
 8. The social robot of claim6, wherein the social expression component comprises the audio-videooutput device.
 9. The social robot of claim 7, wherein the instructionsare further configured for executing a content-selection algorithmincluding selection of the identifier based on the current mental stateindicator that includes at least one of a user mood or a user desire.10. The social robot of claim 1, wherein the instructions are furtherconfigured for configuring the data signal for causing the electronicoutput device to respond, wherein the electronic output device comprisesat least one of a home appliance or motor vehicle and the informationencoded by the data signal comprises operating instructions for the homeappliance or motor vehicle.
 11. The social robot of claim 1, wherein thesocial expression component comprises a face component connected to theat least one circuit, the face component comprising at least twodisplayed facial features controlled by the one or more processors andselected from: a configurable brow, at least one configurable eye, and aconfigurable mouth.
 12. The social robot of claim 2, wherein a biometricsensor is coupled to at least one circuit for providing a data signalresponsive to a bodily state of the user, and wherein the instructionsare further configured for determining the current mental stateindicator further based on the data signal from the biometric sensor.13. The social robot of claim 12, wherein the biometric sensor iscoupled to the at least one circuit via an intervening wireless receiverand transmitter pair.
 14. A method for controlling an environment of asocial robot based on a prior social interaction between the socialrobot and a user thereof, the method comprising: interacting sociallywith the user by operating a social expression component coupled to atleast one circuit for driving components of the social robot andcomprising a display screen configured for displaying facial expressionscontrolled by the at least one circuit according to a social behavioralgorithm, wherein the at least one circuit comprises one or moreprocessors; determining, by the one or more processors, user behaviorbased at least in part on data from a video camera and a microphonecoupled to the at least one circuit, correlating, by the one or moreprocessors, the user behavior to social behavior of the social robot,generating a data signal based on the correlating, comprisingdetermining a current mental state indicator based on the correlating,and causing selection of an identifier of a prerecorded audio-videocontent clip based on comparing semantic metadata associated with theidentifier to the current mental state indicator, and outputting thedata signal configured for causing an electronic output device torespond to information encoded by the data signal.
 15. The method ofclaim 14, further comprising performing the causing selection of anidentifier at least in part by scoring response actions against themental state indicator using a numeric scoring system.
 16. The method ofclaim 15, further comprising transmitting the current mental stateindicator to a remote server and receiving the identifier from theremote server.
 17. The method of claim 15, wherein a memory coupled tothe one or more processors holds identifiers for prerecorded audio-videocontent clips each associated with semantic metadata pertaining tocorresponding ones of the content clips, and further comprisingexecuting a content-selection algorithm, thereby causing the selectionof the identifier.
 18. The method of claim 15, wherein the correlatingfurther comprises determining the current mental state indicatorcomprising at least one of an indicator of a user mood or an indicatorof a user desire.
 19. The method of claim 15, wherein the electronicoutput device comprises an audio-video output device, and furthercomprising configuring the data signal so that the prerecordedaudio-video content clip is output by the audio-video output device. 20.The method of claim 14, wherein the electronic output device comprisesat least one of a home appliance or motor vehicle, and the informationencoded by the data signal comprises operating instructions for the homeappliance or motor vehicle.
 21. The method of claim 15, furthercomprising determining the current mental state indicator further basedon a data signal from a biometric sensor configured to provide the datasignal responsive to a bodily state of the user.